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ABSTRACT 

The  Multi-Spectral  Sensor  Surveillance  System  (M4S)  is  a  multi-year  ONR-sponsored  program  to 
transition  mature  sensor  and  data  fusion  technology  into  existing  and/or  near-future  airborne  surveillance 
platforms.  A  study  phase  and  on-board  sensor  data  fusion  concept-of-proof  demonstration  have  been 
completed  in  1997.  This  paper  describes  the  data  fusion  concepts,  architecture,  and  algorithms  that  have 
been  designed  and  demonstrated  in  these  efforts. 

The  data  fusion  architecture  selected  for  M4S  is  a  distributed  design  in  which  each  on-board  sensor  sub¬ 
system  is  equipped  with  a  single-sensor  tracking  unit  satisfying  all  the  sensor-specific  tracking  needs  in 
addition  to  required  sensor  data  processing  capability.  Thus  scan-to-scan,  or  frame-to-frame  correlation  is 
basically  resolved  on  a  single-sensor  basis,  and  the  outputs  of  each  sensor-subsystem  are  typically  single¬ 
sensor  tracks,  or  tracklets,  i.e.,  stochastically  independent  fractions  of  tracks.  Those  outputs  are  then  fed 
into  a  centralized  multi-sensor,  data  fusion  process  that  performs  track-to-track  association  analysis  and 
fuses  appropriate  single-sensor  tracks  into  multiple-sensor  tracks. 

In  this  way,  each  sensor  sub-system  provides  target  information  complementary  to  each  other  as  well  as 
reinforcing  each  other,  in  terms  of  both  target  identification  and  target  localization,  so  that  the  central  data 
fusion  process  may  produce  a  best  picture  of  each  target  of  interest.  This  system  architecture  also  allows 
each  sensor-specific  tracker  to  temporarily  lose  hold  of  some  targets  but  to  re-acquire  them  later,  yet  to 
maintain  continuous  target  recognition. 

This  data  fusion  process  is  also  connected,  through  an  external  communication  network,  to  off-board 
intelligence  and  surveillance  sources,  such  as  Rivet  Joint,  AW  ACS,  U2,  JSTARS,  etc.,  to  provide  the 
system  with  a  complete  tactical  picture.  This  distributed  architecture  requires  each  on-board  sensor  sub- 
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The  Multi-Spectral  Sensor  Surveillance  System  (M4S)  is  a  multi-year  ONR-sponsored  program  to  transition  mature  sensor  and  data  fusion 
technology  into  existing  and/or  near- future  airborne  surveillance  platforms.  A  study  phase  and  on-board  sensor  data  fusion  concept-of-proof 
demonstration  have  been  completed  in  1997.  This  paper  describes  the  data  fusion  concepts,  architecture,  and  algorithms  that  have  been 
designed  and  demonstrated  in  these  efforts.  The  data  fusion  architecture  selected  for  M4S  is  a  distributed  design  in  which  each  on-board  sensor 
subsystem  is  equipped  with  a  single-sensor  tracking  unit  satisfying  all  the  sensor- specific  tracking  needs  in  addition  to  required  sensor  data 
processing  capability.  Thus  scan-to-scan,  or  frame-to-frame  correlation  is  basically  resolved  on  a  single-sensor  basis,  and  the  outputs  of  each 
sensor- subsystem  are  typically  singlesensor  tracks,  or  tracklets,  i.e.,  stochastically  independent  fractions  of  tracks.  Those  outputs  are  then  fed 
into  a  centralized  multi-sensor,  data  fusion  process  that  performs  track-to-track  association  analysis  and  fuses  appropriate  single-sensor  tracks 
into  multiple- sensor  tracks.  In  this  way,  each  sensor  sub-system  provides  target  information  complementary  to  each  other  as  well  as  reinforcing 
each  other,  in  terms  of  both  target  identification  and  target  localization,  so  that  the  central  data  fusion  process  may  produce  a  best  picture  of 
each  target  of  interest.  This  system  architecture  also  allows  each  sensor-specific  tracker  to  temporarily  lose  hold  of  some  targets  but  to 
re-acquire  them  later,  yet  to  maintain  continuous  target  recognition.  This  data  fusion  process  is  also  connected,  through  an  external 
communication  network,  to  off-board  intelligence  and  surveillance  sources,  such  as  Rivet  Joint,  AW  ACS,  U2,  JSTARS,  etc.,  to  provide  the 
system  with  a  complete  tactical  picture.  This  distributed  architecture  requires  each  on-board  sensor  sub-system  to  perform  more  or  less 
autonomously.  In  order  to  assist  single-sensor  track  initiation  processes,  however,  the  central  data  fusion  process  provides  feedback  tracks  to 
the  sensor  sub-systems,  as  well  as  cross-sensor  or  off-board  cueing  instructions.  Originally,  four  non-developmental  sensors,  ESM,  radar, 
EO/IR,  and  laser  sensors,  have  been  chosen  for  on-board  sensors  that  provide  a  unique  combination  of  electromagnetic  and  optical,  and  passive 
and  active,  sensing.  The  distributed  system  architecture,  however,  allows  us  to  expand  or  modify  this  sensor  suite  without  any  significant 
modification  to  the  system  design,  in  a  more  or  less  plug-n-play  fashion.  The  proof-of-concept  demonstration  was  successfully  completed  in 
1997  showing  the  full  potential  of  the  system  concept,  using  emulated  on-board  sensors  and  off-board  sources.  In  the  succeeding  years  in 
Phase  II,  development  of  concepts  of  on-board,  on-line,  sensor  and  platform  resource  management  will  take  place,  as  well  as  switching 
emulated  sensors  to  real  sensors,  possibly  on  an  airborne  platform. 
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system  to  perform  more  or  less  autonomously.  In  order  to  assist  single-sensor  track  initiation  processes, 
however,  the  central  data  fusion  process  provides  feedback  tracks  to  the  sensor  sub-systems,  as  well  as 
cross-sensor  or  off-board  cueing  instructions. 

Originally,  four  non-developmental  sensors,  ESM,  radar,  EO/IR,  and  laser  sensors,  have  been  chosen  for 
on-board  sensors  that  provide  a  unique  combination  of  electromagnetic  and  optical,  and  passive  and 
active,  sensing.  The  distributed  system  architecture,  however,  allows  us  to  expand  or  modify  this  sensor 
suite  without  any  significant  modification  to  the  system  design,  in  a  more  or  less  plug-n-play  fashion. 

The  proof-of-concept  demonstration  was  successfully  completed  in  1997  showing  the  full  potential  of  the 
system  concept,  using  emulated  on-board  sensors  and  off-board  sources.  In  the  succeeding  years  in  Phase 
II,  development  of  concepts  of  on-board,  on-line,  sensor  and  platform  resource  management  will  take 
place,  as  well  as  switching  emulated  sensors  to  real  sensors,  possibly  on  an  airborne  platform. 


1,0  INTRODUCTION 

A  new  world  order  is  changing  tactical  threat  perspectives  and  operations.  New  pressures  to  minimize 
force  exposure  and  attrition  while  improving  asset  effectiveness  in  limited  conflict  engagements  are  key 
requirements  behind  this  tactical  philosophy.  Because  of  likely  limited  warfare  in  confined  battle  areas, 
situation  awareness,  combat  identification  friend  or  foe,  secure  on  the  move,  and  re-locatable  target 
location  systems  will  be  required  to  support  these  needs.  The  Navy  has  a  pervasive  and  expanded  role  in 
this  new  tactical  scenario.  Littoral  Warfare  and  force  projection  ashore,  in  concert  with  other  force 
structures,  is  vital  to  new  tactical  scenarios.  Shipboard  Surveillance  will  be  the  key  operational  factor  in 
Joint  Mission  Littoral  Warfare. 

A  powerful  approach  to  Littoral  surveillance  is  the  deployment  of  Unmanned  Air  Vehicles  (UAVs)  or 
other  airborne  platforms  equipped  with  multi-spectral  surveillance  capability.  An  aircraft,  launched  from 
land  or  a  Carrier  Battle  Group,  provides  critical,  real-time  intelligence  information  necessary  for  target 
surveillance,  location,  identification,  battle  planning  and  post-strike  Battle  Damage  Assessment  (BDA). 
Ligure  1.1  depicts  the  concept  of  operations  for  a  Multi-Spectral  Sensor  Surveillance  System  (M4S).  In 
this  scenario  a  single  air  vehicle  outfitted  with  multi-spectral  sensors  is  able  to  detect  critical  targets 
through  fusion  of  contact  reports  received  from  off-board  sensors  and  its  own  GMTI  Radar.  Armed  with 
this  information  the  vehicle  and  its  sensors  are  tasked  to  further  classify  and  identify  these  targets  utilizing 
its  short-range  sensors.  Lusing  the  information  from  both  the  long  and  short-range  sensors  allows  the 
M4S  to  continue  to  maintain  track  on  these  critical  targets  even  at  long  range. 
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Figure  1-1 

M4S  Concept  of  Operations 


To  demonstrate  this  capability,  ONR  has  sponsored  a  multi-year  program  to  transition  notional  sensors 
and  data  fusion  technology  into  existing  or  near-future  airborne  surveillance  platforms.  The  objective  of 
this  program  is  to  prove  the  ability  of  enhanced  data  fusion  algorithms  to  detect,  identify,  and  track  targets 
with  improved  detection  probability  and  greater  position  and  identification  accuracy  in  a  reduced  amount 
of  time  with  fewer  false  alarms  using  multiple  dissimilar  on-board  sensors  and  off-board  information. 


The  team  of  ONR,  Litton  Applied  Technology,  Raytheon  Systems  Company,  QuesTech,  CSCI,  and 
SPAWAR  Systems  Center  has  completed  a  study  phase  and  on  and  off-board  sensor  data  fusion  concept- 
of-proof  demonstration.  The  primary  technical  objective  of  this  team  is  to  develop  a  design  architecture 
and  enhanced  data  fusion  algorithms  that  support  the  goal  of  critical  target  identification  in  a  reduced 
time-line. 


The  data  fusion  architecture  selected  for  M4S  is  a  distributed  design  in  which  each  on-board  sensor  sub¬ 
system  is  equipped  with  a  single-sensor  tracking  unit  satisfying  all  the  sensor-specific  tracking  needs  in 
addition  to  required  sensor  data  processing  capability.  Thus  scan-to-scan,  or  frame-to-frame  correlation  is 
basically  resolved  on  a  single-sensor  basis.  These  outputs  are  then  fed  into  a  centralized  multi-sensor, 
data  fusion  process  that  performs  track-to-track  association  analysis  and  fuses  appropriate  single-sensor 
tracks  into  multiple-sensor  tracks.  In  this  way,  each  sensor  sub-system  provides  target  information  that  is 
both  complementary  and  reinforcing  to  the  others  so  that  the  central  data  fusion  process  may  produce  a 
best  picture  of  each  target  of  interest.  This  system  architecture  also  allows  each  sensor-specific  tracker  to 
temporarily  lose  hold  of  some  targets  yet  the  overall  system  can  maintain  continuous  target  recognition. 


This  data  fusion  process  is  also  connected,  through  an  external  communication  network,  to  off-board 
intelligence  and  surveillance  sources,  such  as  Rivet  Joint,  AW  ACS,  U2,  JSTARS,  etc.,  to  provide  the 
system  with  a  complete  tactical  picture.  This  distributed  architecture  requires  each  on-board  sensor  sub¬ 
system  to  perform  more  or  less  autonomously.  In  order  to  assist  single-sensor  track  initiation  processes, 
however,  the  central  data  fusion  process  provides  feedback  tracks  to  the  sensor  sub-systems,  as  well  as 
cross-sensor  or  off-board  cueing  instructions. 

Four  notional  sensors,  ESM,  radar,  EO/IR,  and  laser  sensors  have  been  chosen  for  on-board  sensors  that 
provide  a  unique  combination  of  electromagnetic  and  optical,  and  passive  and  active,  sensing.  The 
distributed  system  architecture,  however,  allows  us  to  expand  or  modify  this  sensor  suite  without  any 
significant  modification  to  the  system  design,  in  a  more  or  less  plug-n-play  fashion. 

The  M4S  proof-of-concept  demonstration  was  successfully  completed  in  1997  showing  the  full  potential 
of  the  system  concept,  using  emulated  on-board  sensors  and  off-board  sources.  In  the  coming  years, 
development  of  concepts  for  on-board,  on-line,  sensor  and  platform  resource  management  will  take  place, 
as  well  as  switching  emulated  sensors  to  real  sensors,  with  the  goal  of  demonstrating  M4S  on  an  airborne 
platform. 


2.0  DISTRIBUTED  DATA  FUSION  ARCHITECTURE  FOR  M4S 

The  choice  between  a  distributed  or  centralized  architecture  is  one  of  the  most  important  design  decisions 
to  be  made  in  the  development  of  any  dissimilar  source  data  fusion  system.  A  distributed  fusion 
architecture  is  characterized  by  a  number  of  parallel  processing  channels  each  of  which  deals  with  a 
subset  of  the  fusion  problem.  These  processing  channels  are  then  combined  to  generate  an  integrated 
track  database.  A  centralized  fusion  architecture  is  essentially  the  opposite  -  the  entire  fusion  problem  is 
dealt  with  as  a  single  entity.  Each  of  these  alternatives  offers  specific  advantages  and  disadvantages,  and 
the  selection  requires  a  careful  consideration  of  the  intended  application  and  the  nature  of  the  input  sensor 
reports. 

As  described  in  the  previous  section,  the  objective  of  M4S  is  to  prove  the  ability  of  enhanced  data  fusion 
algorithms  to  detect,  identify,  and  track  targets  with  improved  detection  probability  and  greater  position 
and  identification  accuracy  in  a  reduced  amount  of  time  using  multiple  dissimilar  on-board.  Eor  M4S  we 
did  not  limit  ourselves  to  the  exact  capabilities  of  existing  sensors,  but  considered  the  probable  evolution 
of  sensor  capabilities  to  the  year  2005.  The  specific  sensors  types  considered  include  a  multi-mode  radar, 
a  infrared  sensor,  an  ELINT  system,  and  a  continuous  wave  laser  capable  of  detecting  vehicle  vibrations. 
Additionally,  we  assumed  the  availability  of  off-board  sensor  data  provide  by  over  communication  links 
to  the  M4S  platform. 

The  notional  radar  is  similar  to  a  potential  variant  of  the  Raytheon  AN/APS- 137B  (V)  5  and  is  capable  of 
operating  in  the  following  modes: 

GMTI  (Ground  Moving  Target  Indicator)  for  the  detection  of  moving  ground  targets, 

PPI  (Plan  Position  Indicator  mode)  for  the  detection  of  ships  and  coast  lines, 

SAR  (Synthetic  Aperture  Radar)  for  imaging  and  classifying  fixed  objects,  and 
Inverse-  Synthetic  Aperture  Radar  (ISAR)  for  imaging  moving  objects  such  as  ships. 

The  notional  IR  sensor  has  capabilities  similar  to  the  Raytheon  ELIR  developed  for  the  LAMPS 
helicopter.  It  is  capable  of  operating  in  both  a  wide  field  of  view  for  search  and  detection,  and  narrow 
field  of  view  for  target  classification. 


The  assumed  ESM  system  for  M4S  is  essentially  the  PLAID  system  being  developed  by  Litton  Applied 
Technology  for  Wright  Laboratories.  Similarly,  the  LASER  is  similar  to  a  system  developed  by  Litton 
Laser  Systems. 

The  specific  features,  reported  by  these  hypothesized  sensor  systems,  are  listed  in  detail  in  Table  2.1.  The 
selected  sensor  systems  are  highly  complimentary  in  nature  and  there  is  significant  potential  for  synergy. 
Lor  example,  there  is  a  mixture  of  active  (RADAR  and  LASER)  systems  and  passive  (ESM  and  LLIR) 
systems.  The  detection  range  varies  from  an  assumed  1000  nm  for  off-board  reporting,  to  80  nm  from 
ESM,  to  2  nm  for  the  LASER,  and  the  notional  data  rates  vary  from  almost  two  hundred  report  per  minute 
to  about  one  per  minute. 


Sensor 

Geo-positionai 
data  reported 

Reported  Features 

Range  at 
5000’  Alt 

Nominal  Data 
Rate 

Comments 

Off-Board 

Sensor 

Data 

Time  of  Intercept 
(TOI) 

Latitude/Longitude 
Measurement  Error 

Emitter  identification 
(ELNOT) 

Emitter  parameters 
(PRI,  RF,  PW,SP) 
Discrete  Attributes  (e.g.. 
Name,  WACBE, 
platform  type,  etc.) 

1000  nm 

180  intercepts/ 
minute 

Wide  area  surveillance 
data  broadcast  by  a 
collection  of  remote 
platforms.  A  mixture  of 
various  sensor  systems 
feed  this  broadcast. 

ESM 

Time  of  Intercept 
(TOI) 

Latitude/Longitude 
Measurement  Error 

Emitter  identification 
(ELNOT) 

Emitter  parameters 
(PRI,  RF,  PW,SP) 

80  nm 

180  intercepts/ 
minute 

Requires  target 
emission. 

Radar  in 

GMTI 

mode 

TOI  Sensor 
location 

Slant  range, 
azimuth 

Velocity  vector 
Measurement  Error 

Sensor  assigned  track 
number 

Wheeled  /  Tracked 
discrimination 

Aircraft  propulsion 
discrimination 

75  nm 

90  tracks  / 
minute 

Assume  1%  false  target 
rate 

Assume  aberrancy  rate 
of  1% 

Assume  high  track 
fragmentation 

Radar  in 
PPi  mode 

as  above 

Sensor-assigned  track 
number 

75  nm 

90  tracks  / 
minute 

as  above 

Radar  in 
SAR/ 
iSAR 
mode 

as  above 

Sensor-assigned  track 
number 

Target  type 

identification  with  prob. 
Image,  Size,  Shape, 
etc. 

65  nm 

1  track/ 
minute 

Assume  no  false  targets 

iR  Search 
Mode 

TOI,  Sensor 
location 

Azimuth 

Measurement  Error 

Sensor-assigned  track 
number 

Target  type 

identification  with  prob. 
Image,  Size,  Shape, 
etc. 

10  nm 

1 0  reportable 
object  /  minute 

Sensor  processing 
determines  reportable 
objects 

Assume  wide  Field  of 
view 

iR 

Ciassifica 
tion  Mode 

as  above 

as  above 

10  nm 

1  reportable 
object  /  minute 

As  above. 

Laser 

Vibration 

TOI,  Sensor 
location 

Slant  range, 
azimuth 

Velocity  vector 
Measurement  Error 

Sensor  assigned  track 
number 

Target  type 

identification  with  prob. 
Vibration  frequency 
meas. 

2  nm 

1  track/ 
minute 

Assume  slewed  with  IR 
sensor  in  Classification 
mode 

Table  2.1 

Assumed  Sensor  Reporting  Characteristics  for  M4S  Sensors 


Additionally,  the  reported  features  are  highly  dependent  on  the  specific  sensor  and  mode,  and  include 
only  the  presence  of  a  target  (RADAR  in  PPI  mode),  images  (SAR,  IRAR,  and  FLIR),  ESM  parametrics, 
and  acoustic  vibration  signature  data. 

For  the  M4S  program,  we  considered  both  the  distributed  and  centralized  candidate  fusion  architectures 
indicated  in  Figure  2.1.  The  centralized  fusion  architecture  has  three  logical  components:  a  fusion  engine, 
a  situation  understanding  function,  and  a  resource  manager.  The  fusion  engine  takes  all  incoming  sensor 
reports,  combines  them  into  tracks,  and  draws  inferences  on  these  tracks.  The  situation  awareness 
function  evaluates  aggregations  of  individual  tracks,  and  attempts  to  understand  the  significance  of  the 
state  and  location  of  the  aggregate  of  tracks.  The  resource  manager,  in  contrast,  exploits  the  knowledge 
captured  by  the  fusion  engine,  and  tasks  the  individual  sensors  available  to  the  system  to  achieve  and 
maintain  tracks. 
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Centralized  Fusion  Architecture 


Distributed  Fusion  Architecture 

Figure  2.1 

Centralized  and  Distributed  Fusion  Architectures  considered  for  M4S 


In  the  distributed  alternative,  the  fusion  engine  is  split  into  a  series  of  single  sensor  target  trackers,  and  the 
multi-sensor  track  fuser.  Thus,  four  sequential  processing  steps  perform  differing  levels  of  fusion:  the 
sensor  target  tracker,  the  track  fuser,  situation  understanding,  and  resource  management.  These  functions 
roughly  correspond  to  the  definition  for  Levels  1  through  4  data  fusion  as  defined  by  the  Joint  Director's 
of  Laboratories  Data  Fusion  sub  panel. 

The  sensor  target  trackers  correlate  information  provided  by  a  single  sensor  or  a  collection  of  similar 
sensors.  To  do  this,  the  sensor  target  tracker  exploits  specific  features  generated  by  a  sensor,  including 
position,  to  construct  pure  output  target  tracks.  For  example,  for  the  ESM  sensor  target  tracker,  the 
information  specific  features  include  those  listed  in  Table  2-1:  RF,  PRI,  SP,  and  PW.  Since  each  sensor 
target  tracker  can  exploit  similar  features  to  aid  in  it  correlation  decision,  it  reliability  is  significantly 
enhanced. 


The  output  of  this  stage  is  a  “tracklet.”  The  notion  of  a  tracklet  is  an  essential  feature  of  the  distributed 
architecture.  Tracklets  are  small,  stochastically  independent  portions  of  a  track  made  up  of  a  sequence  of 
contact  reports  from  single  sensor  (or  sensor  type).  Each  tracklet  has  an  associated  tracker  number 
together  with  whatever  features  have  been  observed  by  the  sensor.  Tracklets  have  temporal  extent,  and 
thus,  a  derived  velocity  component. 

The  second  stage  of  the  distributed  fusion  process  is  the  multi-sensor  track  fuser.  The  multi-sensor  track 
fuser  receives  a  sequence  of  tracklets  from  sensor  target  trackers,  and  correlates  these  tracklets  together, 
largely  because  of  the  track  number  provided  by  the  sensor  target  tracker.  In  this  fashion,  the  fuser 
reconstructs  the  tracks  maintained  by  the  sensor  target  tracker. 


The  challenge  of  the  track  fuser,  then  is  to  determine  when  a  given  target  track  is  represented  in  the  data 
stream  by  more  than  one  target  tracker.  The  sensor  track  fuser  exploits  the  temporal  extent  of  the 
tracklets  to  increase  the  reliability  of  track  correlation.  Feasibility  tables  also  support  the  sensor  track 
fuser.  These  tables  specify  which  combinations  of  sensor  features  are  feasibly  on  platforms  or  objects  of 
interest. 

There  is  some  research  to  indicate  that  a  centralized  fusion  architecture  is  optimal  under  a  number  of 
circumstances.  See  for  example  references  [2]  and  [3].  For  M4S,  however,  we  have  selected  a 
distributed  fusion  architecture.  The  reasons  for  this  decision  are  driven  by  a  number  of  implementation 
considerations.  These  include: 

Computational  resources  are  a  significant  concern  in  the  M4S  environment.  It  is  important  that  the 
system  keep  up  with  the  sensor  data  with  no  significant  data  loss.  By  distributing  the  processing,  the 
problem  of  data  loss  is  effectively  managed.  The  distributed  architecture  allows  us  to  host  the  sensor 
target  trackers  on  separate  processors  from  the  fusion  processor.  Each  of  these  processors  can  be  sized  to 
meet  the  specific  processing  required  of  that  data  path.  Moreover,  the  tracklet  approach  of  inter-processor 
communications  allows  the  fusion  processor  to  query  the  sensor  tracker  for  data  when  it  is  free  to  accept 
more  data.  The  query  -  response  approach  automatically  throttle  the  input  rate  to  the  sensor  fuser  to  the 
rate  at  which  it  can  process  the  data.  In  the  interim,  the  sensor  target  tracker  can  continue  to  extend  the 
temporal  extent  of  tracklets. 

The  distributed  architecture  allows  plug  and  play  capability.  Each  sensor  target  tracker  can  be  designed 
to  exploit  the  special  features  generated  by  each  sensor.  This  capability  allows  for  the  implementation  of 
algorithms  with  very  effective  correlation  of  similar  source  data.  Moreover,  each  sensor  target  tracker 
can  convert  the  input  sensor  stream  into  a  standard  tracklet  format  accepted  by  the  sensor  fuser.  The 
query  response  approach  automatically  prevents  the  sensor  fuser  from  being  saturated  with  data  provided 
by  the  new  feed.  Thus,  a  new  source  of  information  can  be  effectively  added  to  the  sensor  fuser  in  a  plug 
and  plan  fashion. 


3,0  SENSOR  TRACK  SOFTWARE 

As  mentioned  above,  each  sensor  sub-system  has  its  own  sensor  signal  processor  and  a  single-sensor 
tracker,  as  well  as  necessary  sensor  control  mechanism.  As  shown  in  Eigure  3.1,  a  rather  standard  or 
traditional  tracking  system  architecture  was  chosen  for  each  single-sensor  tracker.  This  software 
processes  a  frame  (or  scan)  of  sensor  detections  to  maintain  single-sensor  tracks  and  outputs  those  tracks, 
judged  to  be  sufficiently  mature,  to  the  multi-sensor  data  fusion  process. 


Figure  3-1 

Single-Sensor  Tracker  Software 


Each  frame  of  detections  (or  reports)  from  the  sensor  signal  processor  is  input  into  the  frame-to-track 
correlator  to  obtain  the  frame-wise  optimal  report-to -track  correlation  (association).  For  this  purpose,  the 
JVC  assignment  algorithm,  known  to  be  the  most  efficient,  is  used.  The  track  update  process  updates  the 
associated  report-to-track  pairs  and  unassociated  reports  are  fed  into  the  track  initiation  process.  The 
track  initiation  algorithm  is  designed  to  be  generally  multi-level  and  multi-step.  The  track  termination 
process,  another  standard  component,  is  also  used  to  terminate  tracks  that  are  not  updated  more  than  a 
specified  time  period.  The  tracks  that  are  in  the  track  database  and  that  are  judged  to  be  “mature”  tracks 
by  the  track  monitor  process  will  be  fed  into  the  multi-sensor  data-fusion  process.  In  order  to  alleviate  the 
track  initiation  function,  multi-sensor  tracks  may  be  fed  back  to  each  single-sensor  tracker  as  necessary. 

The  tracks  that  are  judged  to  be  “mature”  enough  are  output  to  the  multi-sensor,  data-fusion  process, 
through  a  track  monitor.  When  a  track  has  accumulated  a  large  number  of  reports  (or  measurements) 
from  each  sensor,  it  is  not  efficient  to  move  the  whole  track  as  a  set.  An  alternative  is  to  represent  the 
track  by  the  target  state  distribution  at  the  most  recent  update  time.  By  doing  so,  we  reduce  the  necessary 
internal  communication  bandwidth  significantly,  but,  at  the  same  time,  we  may  loose  crucial  information 
necessary  to  fuse  information  from  multiple  sensors.  Therefore,  a  compromise  was  made  by  aggregating 
information  contained  in  an  appropriate  number  of  consecutive  sensor  reports.  Those  subsets  of  sensor 
reports  are  called  tracklets,  as  a  small  aggregation  of  temporally  local  sets  of  sensor  reports.  The  length 
of  each  tracklets  can  be  determined  by  the  track  monitor  of  each  sensor  sub-system  based  on  the  sensor 
revisit  rate,  the  target  dynamics,  and  designed  internal  communication  bandwidth.  In  other  words,  these 
tracklets  have  been  chosen  as  a  primary  unit  of  communication  among  all  the  sub-systems  within  the 
overall  data  fusion  systems. 

Each  tracklet  is  an  aggregation  of  consecutive  sensor  reports.  There  are  several  approaches  to  obtain  the 
sufficient  statistics  for  each  tracklet.  For  the  M4S  data  fusion  system,  the  decorrelation  approach  was 
selected.  In  this  approach,  tracklet  statistics  are  calculated  from  the  target  state  distribution  at  the  end  of 
the  last  tracklet  and  from  the  end  of  the  current  tracklet.  This  is  illustrated  in  Figure  3.2.  As  you  can  see 
in  this  figure  each  sensor  report  (or  detection)  is  shown  by  a  box,  and  each  target  state  estimate  at  a  given 
time  is  illustrated  as  a  circle  which  is  an  accumulation  of  information  contained  in  all  the  reports  up  to 
that  point.  In  order  to  calculate  the  statistics  of  a  given  tracklet,  we  subtract  the  information  accumulated 
into  the  target  state  distribution  (up  to  the  last  time  a  tracklet  was  produce)  from  the  current  target  state 
distribution.  This  results  in  only  “new”  information  being  added  to  this  track. 
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(i)  Standard  Tracking  (Filtering)  Process 


Prior  Information  Target  state  Distributions 


(ii)  Tracklet  Representation 

Figure  3.2 

Concept  of  Tracklets 

A  target  state  includes  ordinary  position-velocity  geolocational  state.  But  it  may  also  include  other 
components  that  are  required  or  otherwise  beneficial  for  the  purpose  of  single-sensor  tracking.  For 
example,  for  the  EO/IR  sensor  sub-system,  the  size-shape  parameters  of  the  target  images  and/or  the  other 
state  variables  related  to  the  target  classification/recognition,  can  be  included.  As  discussed  later  in  the 
description  of  the  M4S  demonstration,  the  size-related  parameter  states  are  used  to  simulate  non- 
geolocational  data  for  the  imaging  sensors,  such  as  the  SAR/ISAR-mode  radar  and  the  EO/IR  sensor. 
Similarly,  the  measured  vibration  spectral  parameters  are  used  to  simulate  the  laser  sensor’s  non- 
geolocational  measurements. 


4.0  MULTI-SENSOR,  ON-BOARD/OFF-BOARD  DATA  FUSION 

There  are  a  number  of  enabling  algorithms  required  for  the  Multi-Sensor  Track  Fuser  to  perform  its 
function.  These  algorithms  include: 

A  statistical  tracker.  The  tracker,  a  Kalman  Filter  algorithm,  is  required  to  refine  a  track’s  location  at  a 
specified  time,  to  project  the  track  location  into  the  future,  and  to  estimate  the  tracks  historical  location. 
The  tracker  also  estimates  the  velocity  vector  at  an  arbitrary  time,  as  well  provides  the  uncertainty 
(covariance)  in  position  and  velocity.  The  functions  are  required  to  support  correlation  and  track  to  track 
association  decisions.  The  algorithm  is  based  on  the  Maneuvering  Target  Statistical  Tracker  (MTST) 
algorithm  as  described  in  [8].  The  Multi-Sensor  Track  Fuser  processes  the  tracklets  sent  from  the  Sensor 
Tracker  that  is  a  part  of  each  sensor  subsystem. 

Track  correlation.  A  track  correlation  function  is  required  to  combine  tracklets  from  a  given  sensor 
target  tracker  into  logical  tracks.  The  Sensor  Tracker  of  each  sensor  sub-system  performs  single-sensor 
frame-to-frame  correlation  and  the  results  are  provided  as  the  unique  track  number  attached  to  each 
single-sensor  tracklet.  However,  since  many  of  the  sensor  target  trackers  are  assumed  to  generate  high 
fragmentation  rates,  this  function  is  also  required  to  reduce  the  number  of  track  fragments.  The  statistical 
tracker  is  an  important  element  in  track  correlation.  It  determined  when  to  tracklets  or  track  fragments  are 
position  feasible. 


Track-to-Track  association.  This  is  the  fundamental  algorithm  in  the  sensor  track  fuser.  It  determines 
when  two  tracks  from  different  sources  represent  the  same  physical  object.  To  accomplish  this,  the  track- 
to-track  association  algorithm  compares  the  entire  track  estimates  generated  by  the  statistical  tracker.  It 
also  uses  feasibility  tables  employed  in  threat  inferencing  to  determine  which  combinations  of  report 
features  are  feasible.  The  Sensor  Track  Fuser  Process  maintains  those  multi-sensor  fused  tracks  as  a  two- 
level  structure  as  illustrated  in  Figure  4.1. 


Figure  4.1 

Multi-Sensor  Fused  (Integrated)  Tracks  Data  Structure 

A  multi-sensor,  fused  track  consists  of  the  union  of  all  the  tracklets  contained  in  its  component  tracks  as 
shown  in  Figure  4.1.  However,  the  Sensor  Track  Fuser  Process  maintains  both  the  fused  track,  called  an 
integrated  track,  and  its  component  tracks,  as  independent  tracks.  For  example,  suppose  that  a  fused  track 
has  all  the  component  tracks  shown  in  Figure  4.1,  and  then,  that  one  of  the  sensor  subsystem  sends  a  new 
tracklet  into  one  of  the  component  tracks,  say  the  ESM  track  that  tracks  the  first  emitter  of  the  platform 
target  being  tracked.  Then,  the  Sensor  Track  Fuser  updates  the  first  ESM  track  by  this  new  ESM  tracklet, 
and  at  the  same  time,  the  same  tracklet  is  sent  to  the  fused  (integrated)  track  and  is  used  to  update  the 
fused  track.  By  maintaining  this  double  structure,  it  is  able  to  maintain  the  component-wise  picture  as 
well  as  fused  picture  at  all  the  time,  to  enable  flexible  data  fusion  operations  as  necessary.  As  shown  in 
Eigure  4.1.,  in  general,  any  component  track  may  contain  its  fragments  that  are  put  together  by  the  Sensor 
Track  Euser. 

Threat  Inferencing.  The  purpose  of  threat  inferencing  is  to  evaluate  the  collection  of  features,  from  the 
various  dissimilar  sensors  reporting  on  a  track,  to  infer  more  about  the  identity  or  function  of  an  object. 
Threat  inferencing  exploit  tables  that  define  the  observable  features  associated  with  each  reportable  object 
of  interest.  These  feature  can  include  size  and  shape  (which  are  partially  observed  by  imaging  sensors), 
physical  characteristics  (wheeled  or  tracked,  etc.),  and  emissions  (which  are  partially  observed  by  ESM 
and  acoustic  sensors).  These  tables  indicate  which  combinations  of  features  are  feasible  (used  to  track-to- 
track  association)  and  well  as  to  infer  additional  information  about  a  multi-sensor  track. 


Weapon  Systems 


Weapon  System  Threat  Inferencing 


Figure  4.2.  shows  an  example  of  threat  inferencing  that  takes  place  inside  the  Sensor  Track  Fuser.  Figure 
4.2  shows  the  three  weapon  systems  and  the  set  of  emitters  associated  with  each  system.  For  example, 
SA-8  system  has  three  emitters,  Long  Track,  Thin  Skin,  and  Land  Roll.  The  Sensor  Track  Fuser 
maintains  such  relations  in  Weapon  System  Database  so  that,  from  a  given  set  of  emitter  tracks,  for 
example,  from  Long  Track,  Thin  Skin,  and  Fire  Dome,  the  existence  of  Weapon  System  SA-6B  is 
deduced.  The  Radar  subsystem,  using  its  SAR  and  ISAR  mode,  can  provide  image  attribute  data,  as  well 
as  wheeled/tracked  declaration  attributes  from  the  GMTI  mode.  The  EO/IR  subsystem  can  provide 
additional  image  attribute  data,  while  the  Laser  subsystem  can  identify  the  engine  types  from  the  vibration 
spectrum  analysis.  Those  additional  attributes  are  also  used  by  this  threat  inferencing  to  determine  the 
target  type  classification. 


5,0  SYSTEM  DEMONSTRATION  EXAMPLE 

The  M4S  project  utilizes  a  System  Integration  Laboratory  (SIL)  to  provide  an  effective  test,  evaluation 
and  demonstration  environment.  The  M4S  SIL  is  constructed  of  eight  separate  high  performance  PCs 
running  Microsoft  Windows  NT,  which  are  connected  together  through  an  Ethernet  network.  Each  PC  is 
responsible  for  providing  specific  M4S  functionality  by  hosting  individual  software  components  to  create 
an  integrated  system.  The  SIL  is  configured  with  four  PCs  housing  specific  sensor  subsystems  (ESM, 
Radar,  FLIR/EO,  Laser),  an  off-board  simulator  PC,  a  system  control  PC,  a  multi-sensor  data  fusion  PC, 
and  a  support  PC  which  provides  platform  and  target  generation,  resource  management  activities  and  a 
multipoint  messaging  service.  The  SIL  configuration  is  illustrated  in  Figure  5.1. 
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Figure  5-1 

M4S  Demonstration  Configuration 


The  M4S  system  has  been  developed  to  support  an  improvement  in  the  success  for  tactical  and 
surveillance  missions  through  the  process  of  multi-sensor  data  fusion  and  a  closed  loop  mode  of 
operation.  Scenarios  have  been  developed  to  measure  the  M4S  system’s  effectiveness  for  situations  such 
as  ocean  surveillance,  armored  column,  and  site  assessment  activities.  Due  to  the  flexibility  of  the 
architecture,  M4S  supports  a  “plug  and  play”  type  capability  to  allow  for  a  suite  of  sensors  that  will 
provide  effective  land,  afloat  and  air  coverage.  The  M4S  demonstration  system  is  illustrated  in  Figure 
5.2. 
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Figure  5.2 

M4S  Demonstration  System 


The  M4S  ocean  surveillance  scenario  starts  with  the  M4S  platform  beginning  it’s  flight  from  the  United 
Arab  Emirates  (UAE)  on  a  mission  to  detect,  classify  and  localize  the  position  of  all  surface  traffic  in  the 
Strait  of  Hormuz.  Slightly  after  takeoff,  the  on-board  ESM  sensor  detects  ten  different  radars  operating  in 
the  South  East  portion  of  the  strait.  Through  the  first  stages  of  data  fusion,  M4S  performs  track-to-track 
associations  to  link  associated  emitters  that  belong  to  common  platforms  together,  reducing  the  situation 
to  five  unidentified  tracks.  The  five  tracks  are  determined  to  be  categorized  as  two  hostile  and  three 
unknown  ships.  The  M4S  platform  is  then  directed  North  by  resource  management  activities  to  provide 
ample  sensor  coverage  by  the  on-board  radar  sensor.  The  ISAR  mode  is  used  to  acquire  the  two  hostile 
ships  in  an  attempt  to  identify  their  ship  type.  Eigure  5.3  shows  a  visual  presentation  from  the  multi¬ 
sensor  fusion  application  at  the  intial  stage  of  the  ocean  surveillance  scenario. 


Figure  5.3 

Ocean  Surveillance  Scenario  Start 

After  a  successful  classification  of  two  hostile  cruiser  class  ships  determined  from  the  fusion  of  the  ESM 
and  ISAR  detections,  the  M4S  platform  performs  an  assessment  of  the  situation  to  determine  the  next 
course  of  action.  Realizing  the  tracks  that  have  yet  to  be  identified  and  the  suite  of  sensors  M4S  has 
available,  the  platform  is  directed  to  fly  East  first  for  positioning  and  then  South  to  intercept  the  course  of 
the  nearest  unidentified  track.  Using  the  ELIR/EO  and  Laser  sensors  a  hostile  patrol  boat  is  both 
identified  and  pinpointed  for  position  through  the  use  of  multi-sensor  data  fusion.  The  combination  of 
ESM  detected  emissions,  ELIR/EO  imagery  features  and  Laser  detected  vibration  frequencies  are 
associated  to  provide  a  high  confidence  classification  as  well  as  an  accurate  position  with  very  little 
uncertainty.  The  radar  is  allocated  in  a  PPI  search  mode  to  maintain  track  on  the  hostile  patrol  boat  while 
an  assessment  of  the  situation  is  perfomed  to  direct  the  platform  towards  the  next  unidentified  track  by  the 
resource  manager.  Eigure  5.4  shows  an  example  of  a  cruiser  class  ship  with  fused  information  from  the 
on-board  ESM,  radar,  ELIR/EO  and  Laser  sensors. 


Figure  5.4 

Multi-sensor  Fused  Track 

Situation  assessment  and  resource  management  logic  provides  for  automated  platform  flight  rerouting  and 
dynamic  sensor  allocation  during  the  mission.  The  platform  is  therefore  able  to  continue  in  this  mode  of 
operation  autonomously  due  to  the  M4S  closed  loop  architecture  to  successfully  complete  it’s  mission. 
The  ocean  surveillance  scenario  ends  with  the  M4S  platform  providing  accurate  tracking  and  classifcation 
of  five  separate  tracks:  two  hostile  cruisers,  one  hostile  patrol  boat  and  two  neutral  oil  tankers.  The  two 
following  figures  show  visually  the  difference  between  a  scenario  that  contains  the  M4S  level  of  multi¬ 
sensor  data  fusion  and  a  scenario  that  doesn’t.  Figure  5.5  shows  a  fused  picture  with  five  well  identified 
tracks  with  little  uncertainty  of  their  position. 


Figure  5.5 

Fused  Ocean  Surveillance  Scenario 


Unlike  Figure  5.5,  Figure  5.6  illustrates  a  picture  that  contains  twenty-seven  different  tracks  with  a  high 
level  of  uncertainty  respresented  by  the  90%  containment  ellipses  that  are  drawn  for  each  track. 


Figure  5.6 

Nonfused  Ocean  Surveillance  Scenario 


6.0  YEAR-TWO  EXTENSIONS 

The  next  step  in  the  proof  of  concept  process  for  M4S  will  be  to  concentrate  efforts  in  closing  the 
communications  loop  between  the  data  fusion  process  and  the  sensors.  In  support  of  this,  the  M4S  team 
will  develop  algorithms  and  techniques  for  improved  Situation  Awareness  and  Resource  Management. 
Multi-sensor  data  fusion  results  coupled  with  area  specific  Electron  Order  of  Battle  (EOB)  data  will  be 
used  to  create  a  tactical  picture  that  will  drive  resource  (air  vehicle  and  sensors)  management  decisions. 
Figure  6.1  captures  a  scenario  in  which  five  tanks  and  two  SA8s  are  identified  by  the  sensors  and  the  data 
fusion  process  in  an  area.  Further  analysis  of  this  data  and  the  EOB  results  in  an  aggregated  Situation 
Awareness  picture  showing  their  area  of  influence.  Given  this  information  we  may  assume  a  level  of  air 
coverage  associated  with  this  column  of  tanks  that  may  preclude  the  system  from  collecting  closer  range 
sensor  information  in  this  area.  The  resource  management  functions  could  then  task  the  sensors  to  collect 
data  on  other  targets  of  interest. 
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Figure  6-1 

Situation  Awareness  and  Resource  Management  Concepts 

By  closing  the  loop  and  developing  the  capability  to  task  the  sensors,  we  can  further  identify  the  best 
usage  of  our  notional  sensors  to  further  improve  the  systems  ability  to  accurately  identify  targets  of 
interest  in  a  shorter  time-line. 

With  the  completion  of  the  closed  loop  M4S  architecture,  further  laboratory  testing  can  be  done  to 
measure  the  effectiveness  of  the  correlation  and  sensor  tasking  algorithms.  While  the  M4S  SIL  can 
continue  to  be  used  to  test  various  combinations  of  sensors,  the  real  test  will  be  to  interface  M4S  will  real 
sensors  on  an  airborne  platform.  The  MOE  analysis  collected  from  the  first  year  of  testing  is 


encouraging.  Completing  the  architecture  by  adding  Situation  Awareness  and  a  Resource  Manager 
should  further  serve  to  improve  the  systems  ability  to  quickly  and  efficiently  localize  and  identify  tracks. 
Armed  with  this  data,  evaluating  M4S  on  an  airborne  platform  will  continue  to  show  how  current  and 
near-term  sensor  systems  can  be  made  even  more  efficient  and  useful  without  any  hardware 
modifications.  Coupling  these  sensor  systems  with  M4S  will  not  only  allow  us  to  make  better  use  of 
current  systems  but  will  aid  our  forces  in  more  effectively  locating  and  identifying  critical  targets. 
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